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Problem 


Extensive  studies  have  been  made  of  the  effect  of  practice  or  adjunct  questions  on 
learning.  Generally,  findings  have  indicated  that,  under  the  proper  conditions,  they  can 
facilitate  learning  of  material  similar  to  that  covered  by  the  practice  questions  and 
material  unrelated  or  incidental  to  that  covered  by  the  questions.  Because  of  the 
laboratory  controls  employed  in  these  studies,  however,  the  experimental  environment  is 
quite  different  from  that  of  a real-world  classroom  situation. 

Purpose 


The  purpose  of  this  research  and  development  was  to  evaluate  the  effect  of  practice 
questions  in  a real-world  training  environment  in  which  the  instructional  materials  were 
designed  to  teach  students  to  remember  specific  facts  and  procedures. 

Approach 

The  subjects  were  male  students  enrolled  in  the  Circuits  II  course  at  the  Navy's 
Interior  Communications  "A"  School  from  November  1977  through  May  1978.  This  course 
is  designed  to  teach  students  the  factual  and  procedural  information  prerequisite  to  their 
job.  The  instruction,  therefore,  is  memory  rather  than  performance  oriented.  They  were 
assigned  to  one  of  three  experimental  groups  or  to  a control  group.  Members  of  the  first 
experimental  group  received  a workbook  in  which  all  questions  were  identical  to  those  in 
lesson  tests  or  the  final  test;  the  second  group,  in  which  half  the  questions  were  identical; 
and  the  third  group,  in  which  none  of  the  questions  were  identical.  The  control  group 
received  no  workbook.  Although  members  of  the  experimental  groups,  hereafter  referred 
to  as  Groups  ALL,  HALF,  and  NONE,  could  use  the  workbook  in  any  way  they  chose,  they 
were  directed  to  the  book  to  answer  questions  after  completing  a portion  of  the  lesson 
materials.  Thus,  they  could  become  aware  of  the  degree  of  similarity  between  workbook 
and  test  items. 

At  the  end  of  the  study,  the  four  groups  were  compared  on  the  following  variables: 
performance  on  the  final  test  and  on  subtests  made  up  of  items  within  the  test;  lesson  test 
scores;  time  required  to  complete  the  course;  and  total  number  of  tries  on  lesson  tests. 

Results 


1.  Group  ALL  students  performed  significantly  better  than  the  other  students  on 
the  final  test.  There  was  little  difference  in  the  performance  of  the  other  three  groups, 
even  though  those  in  Group  HALF  had  more  exposure  to  the  final  test  questions. 

2.  Control  group  students,  who  had  no  workbook,  scored  significantly  higher  on 
items  seen  once  before  (in  lesson  tests)  than  Group  HALF  and  NONE  students,  and  Group 
ALL  students  scored  higher  than  Group  HALF  students  on  items  seen  twice  before  (in 
workbooks  and  lesson  tests).  There  was  no  significant  difference  in  performance  of  Group 
HALF,  Group  NONE,  and  control  group  students  on  items  not  seen  prior  to  the  final  test. 
These  items  are  comparable  to  "incidental"  questions  in  the  typical  adjunct  question 
study. 


jtftflMHttBflHifeMMtMMMiaMMiflBiMiaHMMflMattiaiaMafc-  ^553KShSSSSi2||Shhb25II ^ 


3.  Group  ALL  students  performed  significantly  better  than  the  others  on  the  lesson 
tests,  followed  by  those  in  Group  HALF.  There  was  little  difference  in  the  performance 
of  Group  NONE  and  control  group  members. 

4.  Group  ALL  students  required  the  least  time  to  complete  the  course,  followed  by 
those  in  Group  HALF,  Group  NONE,  and  the  control  group.  They  also  had  the  fewest 
number  of  tries  on  lesson  tests,  followed  by  those  in  Group  HALF,  the  control  group,  and 
Group  NONE. 

Conclusions 

In  a real-world  learning  environment,  practice  questions  can  have  a detrimental 
effect  on  learning  factual  and  procedural  information  incidental  to  the  information 
covered  by  the  practice  questions.  Further,  if  they  are  not  related  to  the  test  questions, 
both  test  performance  and  study  time  are  adversely  affected. 

Recommendations 

1.  If  the  intent  of  the  instruction  is  to  have  students  remember  important  pieces  of 
factual  and  procedural  information  (as  in  the  present  study),  a practice  item  and  a test 
item  should  be  individual  for  each  piece  of  information.  There  should  be  no  practice  on 
information  that  will  not  be  tested.  If  time  and  resource  constraints  do  not  permit  all 
important  information  to  be  tested,  the  course  material  should  be  prioritized  so  that  the 
most  crucial  information  can  be  practiced  and  tested.  The  same  prescription  applies  to 
classroom  instruction  where  the  teacher  provides  practice  by  oral  quizzing  or  in 
workbooks,  and  to  other  situations  in  which  information  is  to  be  memorized.  If  the  intent 
of  the  educational  program  is  to  transfer  learning  to  new  situations,  however,  practice 
items  would  necessarily  not  be  the  same  as  test  items. 

2.  Future  research  on  practice  questions  should  be  accomplished  in  an  ongoing 
training  program,  and  should  include  an  investigation  of  the  effects  of  different 
instructional  contents,  question  types,  and  question  frequency. 
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INTRODUCTION 


Problem 


The  Navy  Instructional  Program  Development  Centers,  under  the  Chief  of  Naval 
Education  and  Training,  are  tasked  with  analyzing,  designing,  and  developing  a large 
portion  of  the  Navy's  technical  training  courses.  The  development  effort  includes  writing 
practice  or  self-study  questions  to  accompany  the  instructional  materials.  Educational 
psychologists  have  studied  the  effects  of  practice  or  adjunct  questions  extensively1  and 
have  concluded  that,  under  the  proper  conditions,  they  can  facilitate  learning.  Because  of 
the  laboratory  controls  employed  in  these  studies,  however,  the  experimental  environment 
is  quite  different  from  that  in  a real-world  classroom  situation. 

Purpose 

The  purpose  of  this  research  and  development  was  to  evaluate  the  effects  of  practice 
questions  in  a real-world  training  environment.  The  findings  are  intended  to  be  used  by 
the  Instructional  Program  Development  Centers  in  formulating  guidelines  for  writing 
practice  questions. 

Background 

A considerable  number  of  studies  have  investigated  the  effects  of  practice  or  adjunct 
questions  on  learning.2  Typically,  in  these  studies,  the  performance  of  one  or  more 
experimental  groups  is  compared  to  that  of  a control  group.  All  subjects  in  all  groups  are 
given  a passage  of  instructional  material.  Those  in  the  experimental  groups  are  also  given 
a number  of  practice  or  adjunct  questions  in  which  the  variables  can  be  manipulated 
across  groups  in  a number  of  ways.  For  example,  the  questions  may  be  asked  before  or 
after  subjects  have  read  the  passage,  they  may  be  factual  or  inferential,  they  may  or  may 
not  be  similar  or  identical  to  those  included  in  the  final  test,  or  they  may  vary  as  to  the 
number  included  or  the  frequency  of  adminstration.  Experimental  subjects  are  usually 
required  to  answer  the  questions  without  referring  back  to  the  material.  Finally,  all 
groups  are  given  a final  test,  and  the  results  are  compared  to  determine  whether  providing 
adjunct  questions  affected  performance. 


1 Anderson,  R.  C.,  <5c  Biddle,  W.  B.  On  asking  people  questions  about  what  they  are 
reading.  In  G.  Bower  (Ed.),  Psychology  of  learning  and  motivation  (Vol.  9).  New  York: 
Academic  Press,  1973. 

2Rothkopf,  E.  Z.,  <5c  Bisbicos,  E.  E.  Selective  facilitative  effects  of  interspersed 
questions  on  learning  from  written  materials.  Journal  of  Educational  Psychology,  1967, 
58,56-61. 


Results  of  such  studies  have  shown  that  providing  practice  questions  after— as 
opposed  to  before— subjects  have  read  the  pertinent  section  of  the  reading  passage  can  be 
effective  in  some  situations.  Generally,  experimental  groups  who  are  given  practice 
questions  after  they  have  completed  the  passage  perform  better  than  a no-question 
control  group  on  final  test  questions  that  are  similar  or  identical  to  the  practice 
questions,  as  well  as  on  those  that  are  unrelated  or  incidental  to  the  practice  questions.3 
This  second  finding  is  important  because  it  shows  that  providing  practice  questions  after 
sections  of  instructional  materials  helps  the  student  to  learn  information  that  is  unrelated 
to  that  covered  in  the  questions.  Presumably,  this  effect  occurs  because  the  practice 
questions  focus  the  student's  attention  on  the  type  of  question  (e.g.,  factual  vs. 
inferential)  and/or  the  type  of  information  (e.g.,  general  vs.  detailed)  that  will  be  in  the 
final  test. 

The  implications  of  these  findings  for  instructional  design  and  development  are  not 
clear,  primarily  because  the  experimental  environment  differs  greatly  from  the  real-world 
individualized  instructional  environment,  where  students  rarely  are  prohibited  from 
reviewing  the  instructional  materials  prior  to  answering  practice  questions.  In  a study  by 
Hiller,4  students  were  permitted  to  check  their  answers  to  practice  questions.  They  were 
required  to  answer  the  practice  questions  before  looking  back,  however,  and  were  not 
allowed  to  erase  any  incorrect  answers.  This  situation  produced  inferior  performance  in 
one  condition  and  no  difference  in  another— relative  to  a control  group— on  questions 
unrelated  to  the  inserted  questions.  Given  these  results,  it  may  be  that  practice  questions 
actually  inhibit  learning  in  an  individualized  learning  environment.  The  present  research 
examines  this  possibility  by  varying  the  number  of  practice  questions  that  are  identical  to 
the  test  questions  in  an  ongoing  technical  training  course. 


3 See  footnote  2. 

“Hiller,  J.  H.  Learning  from  prose  text:  Effects  of  readability  level,  inserted  question 
difficulty,  and  individual  differences.  Journal  of  Educational  Psychology,  1974,  66,  202- 
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METHOD 


Subjects 

The  subjects  re  289  male  students  who  were  enrolled  in  the  Circuits  11  course  at 
the  Navy's  Interior  Communications  "A"  School,  San  Diego,  CA  from  November  1977 
through  May  1978.  The  Circuits  II  course  is  a 1-week,  self-study  unit  comprised  of  seven 
lessons.  After  a student  studies  the  material  for  a lesson,  he  takes  a lesson  test,  which  is 
comprised  of  both  multiple-choice  and  matching  items  and  is  administered  on  a computer. 
The  passing  grade  for  lesson  tests  is  70  percent.  If  the  student  does  not  reach  this 
criterion,  he  must  retake  the  test  or  the  computer.  If  he  fails  after  three  tries,  he  must 
take  a written  test  given  by  the  instructor.  Test  items  are  not  changed  for  retakes  and 
the  student  is  not  given  the  correct  answer.  When  a student  has  successfully  completed 
all  seven  lessons,  he  takes  a final  test  on  the  computer.  This  test  includes  50  items,  some 
of  which  are  identical  to  those  included  in  the  lesson  tests.  The  student  must  answer  64 
percent  of  these  items  correctly  to  complete  the  course. 

Procedure 


Subjects  were  assigned  to  one  of  three  experimental  groups  or  to  a control  group.  All 
of  the  subjects  proceeded  through  the  course  as  described  above.  Those  in  the 
experimental  groups,  however,  received  a workbook  containing  practice  questions  at  the 
beginning  of  the  course.  Those  in  the  control  group  did  not  receive  a work’  ook. 

Although  all  of  the  workbooks  contained  the  same  number  and  type  of  questions,  they 
differed  as  to  the  number  of  questions  included  that  were  identical  to  those  included  in 
the  lesson  tests  or  in  the  final  test.  In  the  workbook  provided  to  the  first  experimental 
group,  all  the  questions  were  identical;  in  that  provided  to  the  second  group,  half  were 
identical;  and  in  that  provided  to  the  third  group,  none  were  identical.  Students  in  the 
three  experimental  groups,  which  will  be  referred  to  hereafter  as  Groups  ALL,  HALF,  or 
NONE,  could  use  the  workuooK  in  any  way  they  chose.  The  instruction  was  designed, 
however,  so  that  they  were  dirf  cted  to  the  workbook  to  answer  questions  after  completing 
a portion  of  the  lesson  materials.  Recause  students  were  tested  after  each  lesson,  they 
had  an  opportunity  to  become  aware  of  the  degree  of  similarity  that  existed  between 
practice  and  test  questions  as  they  progressed  through  the  lesson  materials  and  lesson 
tests.  This  procedure  is  somewhat  different  than  the  procedure  used  in  the  majority  of 
adjunct  questioning  studies,  in  which  students  are  only  given  one  lesson  to  study  and  one 
criterion  test.  In  this  situation,  the  student  does  not  have  the  opportunity  to  learn  the 
relationship  between  the  practice  and  test  questions. 

Although  all  students  in  a given  class  (i.e.,  those  entering  each  week  (about  20))  were 
assigned  to  the  same  group,  group  conditions  were  randomly  assigned  to  classes.  This 
resulted  in  64  students  being  assigned  to  Group  ALL,  80  to  Group  HALF,  84  to  Group 
NONE,  and  61  to  the  control  group. 

Performance  Variables 


The  four  groups  were  compared  on  the  following  variables: 

1.  Final  test  performance,  including  final  test  score  and  performance  on  the 
following  subgroups  of  items: 

a.  Those  that  had  been  seen  once  before  (on  the  lesso.  tests)  by  Group  HALF 
and  NONE  and  by  control  group  subjects.  (Group  ALL  subjects  had  seen  them  twice 
before--once  on  the  lesson  test  and  once  in  the  workbook.) 
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b.  Those  that  had  been  seen  twice  before  (in  the  workbook  and  on  the  lesson 
tests)  by  Groups  ALL  and  HALF. 

c.  Those  that  had  not  been  seen  before  by  Group  HALF,  Group  NONE,  and 
control  group  subjects.  This  is  the  measure  of  performance  on  questions  incidental  to  the 
inserted  questions. 

2.  Scores  for  each  lesson  test  in  the  order  taken. 

3.  Time  required  to  complete  the  course  (in  minutes). 

4.  Total  number  of  tries  on  all  lesson  tests  (maximum  of  three  tries  per  test). 
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RESULTS  AND  DISCUSSION 


Final  Test  Performance 

As  shown  in  Table  I,  Group  ALL  subjects,  who  had  received  workbooks  in  which  all 
questions  were  identical  to  those  in  lesson  tests  or  in  the  final  test,  performed 
significantly  better  on  the  final  test  than  did  other  subjects  (F  (3,285)  = 57.16,  g < .001). 
There  was  no  significant  difference  in  the  performance  of  HALF,  NONE,  and  control 
group  members,  even  though  Group  HALF  subjects  had  been  exposed  to  more  of  the  final 
test  questions  than  had  those  in  the  other  two  groups. 

Final  test  data  were  further  analyzed  by  dividing  test  items  into  subtests,  depending 
on  how  many  times  the  various  groups  had  seen  the  item  prior  to  the  final  test.  The  mean 
performance  of  the  various  study  groups  on  these  subtests  is  included  in  Table  1.  Results 
of  an  analysis  of  variance  performed  on  subject  results  are  described  below: 

1.  Subtest  comprised  of  items  seen  once  before-Control  group  members,  who  had 
no  workbook,  scored  significantly  higher  on  this  subtest  than  did  members  of  Groups 
HALF  and  NONE  (F  (2,212)  = 5.33,  g < .005).  A possible  explanation  for  this  result  is  that 
students  in  the  HALF  and  NONE  groups  learned  that  questions  seen  before  were  not  likely 
to  be  repeated.  Therefore,  they  may  not  have  tried  to  remember  them  or  they  may  not 
have  studied  the  material  related  to  them  while  preparing  for  the  final  test. 

2.  Subtest  comprised  of  items  seen  twice  before— Group  ALL  subjects  performed 
significantly  better  on  this  subtest  than  did  Group  HALF  subjects,  even  though  both 
groups  had  an  equal  amount  of  exposure  to  the  items.  This  result  is  consistent  with  that 
for  the  subtest  comprised  of  items  seen  once  before  and  may  be  explained  in  the  same 
way. 


3.  Subtest  comprised  of  items  only  on  the  final  test— These  items  are  comparable  to 
the  "incidental"  questions  in  the  typical  adjunct  question  study.  There  was  no  significant 
difference  in  the  performance  of  members  of  Group  HALF,  Group  NONE,  and  the  control 
group  on  this  subtest.  Thus,  the  presence  of  practice  questions  did  not  facilitate  the 
performance  of  Groups  HALF  and  NONE  relative  to  that  of  the  control  group. 

Lesson  Test  Scores 


Figure  1 compares  the  mean  performance  of  the  study  groups  on  the  seven  lesson 
tests.  It  should  be  noted  that  subjects  were  not  required  to  study  the  lessons  in  any 
particular  order.  Thus,  one  subject  in  a particular  group  may  have  taken  Lesson  2 first; 
and  another,  Lesson  7.  Since  this  measure  compares  performance  on  the  lesson  tests  in 
the  order  they  were  taken,  data  from  different  tests  can  contribute  to  a given  data  point. 
This  did  not  present  a problem  in  interpreting  results,  however,  since  all  of  the  lessons 
were  independent  in  topic  and  there  were  no  systematic  differences  between  groups  in  the 
order  in  which  lessons  were  completed. 

As  shown  in  Figure  1,  the  performance  of  Group  ALL  subjects  was  superior  to  that  of 
the  other  three.  Group  HALF  subjects  generally  performed  between  than  those  in  Group 
NONE  and  the  control  group.  Both  main  effects  and  the  interaction  are  significant:  F 
groups  (3,285)  = 68.67,  g < .001;  F lessons  (6,1710)  = 28.16,  g < .001;  F interaction 
(18,1710)  = 2.62,  g < .001. 

It  is  interesting  to  note  the  data  for  the  first  lesson.  Here  the  groups  order  as 
follows:  ALL,  HALF,  control,  and  NONE,  with  large  differences  between  the  HALF  and 
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Table  1 
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Figure  1.  Mean  performance  of  ^aidy  groups  on  lesson  tests  in  the  order  taken. 


ALL  groups  and  the  control  and  NONE  groups.  This  result  can  be  explained  by  assuming 
that  the  experimental  groups  began  the  course  using  the  workbook  questions  to  guide  their 
study  behavior.  For  those  in  Group  ALL,  this  was  an  optimal  strategy,  since  their 
workbook  questions  were  identical  to  the  test  questions.  For  those  in  Group  HALF,  only 
half  of  the  workbook  questions  were  the  same,  so  they  did  not  perform  as  well  as  those  in 
Group  ALL.  Those  in  the  control  group  had  no  workbook  and  no  guidelines;  consequently, 
they  apparently  tried  to  study  everything  and  performed  almost  as  well  as  those  in  Group 
HALF.  Finally,  those  in  Group  NONE  found  that  their  guidelines  were  completely 
inaccurate,  and,  not  surprisingly,  they  had  the  lowest  performance.  After  the  first  lesson, 
Group  NONE  subjects  apparently  realized  that  their  workbook  was  misleading  and  they 
began  to  study  everything.  As  a result,  their  performance  became  more  like  that  of  the 
control  group.  It  is  apparent  from  this  analysis,  and  from  the  final  test  data,  that 
practice  questions  play  a major  role  in  shaping  study  behaviors. 

The  general  decline  in  performance  for  all  groups,  particularly  on  the  last  test  taken, 
can  be  attributed  to  two  factors.  First,  subjects  tended  to  leave  the  more  difficult  tests 
until  last.  Second,  the  fact  that  they  were  permitted  to  leave  class  early  on  the  final  day 
if  they  finished  the  final  lesson  and  passed  the  final  lesson  test  produced  a "goal  gradient" 
effect. 

Study  Time  Required 

Based  on  the  assumption  discussed  above  (i.e.,  that  students  begin  the  course  by  using 
their  workbook  and  adjust  their  study  behavior  depending  on  how  much  it  helps  them),  it 
can  be  predicted  that  (1)  Group  NONE  and  control  group  subjects  would  require 
approximately  equal  time  to  complete  the  course,  since  they  would  need  to  study  all  of 
the  material;  (2)  Group  HALF  subjects  would  require  less  time  because  of  their  exposure 
to  more  questions  identical  to  those  in  lesson  and/or  final  tests;  and  (3)  Group  ALL 
subjects  would  require  the  least  amount  of  time,  because  they  had  been  exposed  to  all  of 
the  test  questions.  As  shown  in  Table  1,  group  means,  expressed  in  minutes,  were 
generally  as  predicted.  An  ANOVA  performed  on  study  time  data  showed  a significant 
difference  among  groups:  F (3,285)  = 10.18,  £ < .001. 

Number  of  Attempts  on  Lesson  Tests 

As  indicated  previously,  a student  could  take  each  lesson  test  up  to  three  times  on 
the  computer  if  he  continually  failed.  Thus,  the  number  of  tries  on  lesson  tests  could 
range  from  7 to  21.  Given  the  previous  findings,  the  groups  would  be  expected  to  order  as 
follows  on  this  variable:  Group  ALL,  Group  HALF,  control  group,  and  Group  NONE.  This 
assumption  was  supported,  as  shown  by  the  group  means  in  Table  1.  An  ANOVA 
performed  on  lesson  test  data  showed  a significant  difference  among  groups:  F(3,285)  = 
26.60, £ < .001. 


CONCLUSIONS 


Two  conclusions  can  be  drawn  from  the  results  from  the  final  test  and  the  subtests. 
First,  adjunct  questions  in  this  situation  did  not  help  students  learn  information  incidental 
to  the  adjunct  questions.  Second,  students  in  the  HALF  and  NONE  Groups  apparently 
realized  that  the  questions  in  their  workbook  were  not  likely  to  be  included  in  tests;  thus, 
they  did  not  pay  as  much  attention  to  the  lesson  test  questions  as  those  in  the  ALL  Group, 
who  learned  that  questions  in  their  workbook  would  be  included  in  the  rests,  and  the 
control  group,  who  had  no  workbook  for  guidance.  As  a result,  students  in  the  HALF  and 
NONE  Groups  did  not  perform  as  well  as  the  others  on  questions  identical  to  the  lesson 
test  questions.  Thus,  adjunct  questions  had  an  inhibiting  effect  on  their  performance. 
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RECOMMENDATIONS 


It  is  dear  that  practice  questions  have  important  effects  on  performance,  study 
behavior,  and  study  time.  These  results,  which  confirm  and  extend  Hiller's  (1974)  findings, 
have  direct  implications  for  the  design  and  development  of  individualized  instructional 
materials.  If  the  intent  of  the  instruction  is  to  have  students  remember  important  pieces 
of  information  (as  in  the  present  study),  a practice  item  and  a test  item  should  be 
individual  for  each  piece  of  information.  There  should  be  no  practice  on  information  that 
will  not  be  tested.  If  time  and  resource  constraints  do  not  permit  all  important 
information  to  be  tested,  the  course  material  should  be  prioritized  so  that  the  most 
crucial  information  can  be  practiced  and  tested.  The  same  prescription  applies  to 
classroom  instruction  where  the  teacher  provides  practice  by  oral  quizzing  or  in 
workbooks,  and  to  other  situations  in  which  information  is  to  be  memorized,  if  the  intent 
of  the  educational  program  is  to  transfer  learning  to  new  situations,  however,  practice 
items  would  necessarily  not  be  the  same  as  test  items. 

Future  research  on  practice  questions  should  be  accomplished  in  an  ongoing  training 
program,  and  should  investigate  the  effects  of  different  instruction  content,  different 
question  types,  and  different  question  frequency. 
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